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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


- 1  - 


•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department 
of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  symposium. 

James  B.  Greene  Jr.  Keith  F.  Snider,  PhD 

Rear  Admiral,  U.S.  Navy  (Ret.)  Associate  Professor 
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currently  researches  and  publishes  on  international  economics,  defense  economics,  health 
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Doug  Bodner — Bodner  is  a  senior  research  engineer  in  the  Tennenbaum  Institute  at  the  Georgia 
Institute  of  Technology.  His  research  focuses  on  computational  analysis  and  decision  support  for 
design,  operation,  and  transformation  of  enterprise  systems.  His  work  has  spanned  a  number  of 
industries,  including  automotive,  electronics,  energy,  health  care,  military  acquisition,  paper  and  pulp, 
semiconductors,  and  telecommunications.  He  is  a  senior  member  of  the  Institute  of  Electrical  and 
Electronics  Engineers  (IEEE)  and  the  Institute  of  Industrial  Engineers  (ME),  and  a  member  of  the 
Institute  for  Operations  Research  and  Management  Science  (INFORMS).  He  is  a  registered 
professional  engineer  in  the  State  of  Georgia,  [bodner@gatech.edu] 

Abstract 

Defense  acquisition  is  characterized  by  significant  levels  of  risk  throughout  the  lifecycle.  Risk, 
of  course,  may  result  in  undesirable  outcomes.  Deriving  from  many  sources,  both  technical 
and  organizational,  risk  is  inherently  a  sociotechnical  phenomenon  in  enterprises  such  as 
acquisition.  As  such,  it  is  difficult  to  address.  At  the  same  time,  fiscal  pressures  are  causing 
decreased  funding  and  increased  expectations  for  acquisition  performance.  This  points  to  the 
importance  of  risk  characterization  and  mitigation.  Our  previous  work  has  focused  on  using 
simulation  to  model  and  analyze  acquisition  processes  and  incentives  in  order  to  understand 
how  they  can  be  designed  to  improve  outcomes.  Traditional  simulation  analysis  is  not  well 
suited  to  modeling  the  sociotechnical  complexities  of  risk  in  a  systematic  way,  though.  This 
paper  presents  a  decision/event  network  construct  implemented  within  enterprise  simulation 
models  to  represent  the  complexities  of  risk  over  time.  The  F-35  Joint  Strike  Fighter  program 
is  analyzed  with  respect  to  risk  and  potential  outcomes  using  this  enterprise  simulation 
framework  that  accounts  for  sociotechnical  phenomena.  Risk  mitigation  strategies  are 
identified  and  presented. 

Introduction 

Defense  acquisition  is  characterized  by  significant  levels  of  risk  throughout  the 
lifecycle  of  new  system  development,  production,  and  sustainment.  Fundamentally,  risk 
results  from  the  combination  of  an  uncertain  future  (probabilities)  and  its  potential  bad 
outcomes  (magnitudes).  Classic  risk  drivers  in  acquisition  include  immature  technologies, 
overly  optimistic  baseline  cost  and  schedule  estimates,  overly  stringent  requirements,  poorly 
understood  implicit  requirements,  and  changing  missions  and  environments.  Increasingly, 
risks  come  from  new  sources,  such  as  fiscal  pressures  on  government  spending  and 
transformative  initiatives  in  the  acquisition  enterprise. 

This  paper  presents  an  enterprise  simulation  approach  to  analyzing  risk  in  major 
acquisition  programs.  This  approach  models  interactions  between  forms  and  agencies 
within  the  acquisition  enterprise,  and  it  also  includes  a  decision/event  network  that  is  issued 
to  characterize  the  relationships  of  risk  drivers  to  outcomes.  The  approach  is  used  to  model 
the  acquisition  of  the  F-35  Joint  Strike  Fighter,  a  large-scale  acquisition  program  that  has 
seen  effects  from  risk.  This  program  is  currently  in  a  combination  of  system  design  and 
development,  and  low-rate  initial  production. 

The  remainder  of  this  paper  is  organized  as  follows.  The  next  section  describes 
enterprise  simulation  in  the  context  of  risk  analysis.  Then  the  F-35  program  is  discussed, 
with  a  particular  focus  on  risks  and  current  outcomes.  An  enterprise  simulation  model  of  the 
F-35  program  is  presented  next.  Risks  within  the  model  are  analyzed  and  discussed.  Finally, 
conclusions  and  future  research  are  presented. 
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Simulation  of  Acquisition  Programs 

Computer  simulation  has  traditionally  been  used  to  assess  system  performance 
during  design  and  operation.  One  widespread  use  of  simulation  has  been  in  manufacturing, 
as  a  means  of  predicting  such  metrics  as  throughput  and  cost  for  a  factory  system  (Smith, 
2003).  Simulation  enables  the  analysis  of  behavior  and  performance  overtime,  taking  into 
account  probabilistic  effects  (Law,  2007).  Simulation  can  provide  analysis  under  different 
system  design  and  operating  scenarios,  thus  enabling  possible  system  designs  or 
modifications  to  be  studied  prior  to  adoption. 

In  recent  years,  simulation  has  seen  application  in  a  number  of  additional  acquisition- 
related  areas  beyond  manufacturing,  including  supply  chain  design  and  analysis  (Kleijnen, 
2005),  software  system  development  (Madachy,  2008),  acquisition  processes  (Bodner, 
Rahman,  &  Rouse,  2010;  Ford  &  Dillard,  2009),  and  fleet  sustainment  (Smith,  Searles, 
Thompson,  &  Cranwell,  2006).  While  these  are  important  constituent  elements  of  a  large- 
scale  acquisition  program,  there  are  other  important  elements  to  consider,  such  as  the 
enterprise  nature  of  the  public-private  partnership  that  comprises  the  program.  Increasingly, 
firms  that  develop  and  manufacture  complex  products  and  systems  employ  an  enterprise 
paradigm  that  involves  a  large  number  of  stakeholder  firms  (Bodner  &  Lee,  in  press). 

The  traditional  use  of  simulation  has  been  to  study  the  technical  aspects  associated 
with  acquisition  program  elements.  These  include  work  processes,  quality,  schedules,  part 
flows,  inventory  levels,  bottlenecks,  costs,  and  lead  times.  While  this  type  of  analysis  is  quite 
useful,  it  does  not  capture  the  sociotechnical  aspects  of  the  enterprise,  with  its  multi-actor 
collaborations,  decision-making,  and  risks.  Recently,  two  related  types  of  simulation 
methods  have  come  into  usage — agent-based  simulation  (Hillebrand  &  Stender,  1994; 

Saam  &  Schmidt,  2001)  and  organizational  simulation  (Nissen,  2007;  Prietula,  Carley,  & 
Gasser,  1998;  Rouse  &  Boff,  2005).  Agent-based  simulation  focuses  on  the  interaction  of 
different  agents  in  an  eco-system,  where  agents  can  represent  individuals  or  firms.  For 
example,  it  has  been  used  to  model  supply  chain  actor  interactions  (Albino,  Carbonara,  & 
Giannoccaro,  2007).  Organizational  simulation  seeks  to  model  the  behavior  of  people  and 
firms  in  the  context  of  a  world  model  and  organizational  story.  A  world  model  represents  the 
elements  internal  to  the  organization  being  modeled,  as  well  as  those  relevant  external 
elements.  The  organizational  story  represents  a  scenario  being  modeled.  Organizational 
simulation  has  been  used  to  model  healthcare  delivery,  research  and  development,  and 
electronics  design  (Rouse  &  Bodner,  2009).  These  two  types  of  simulation  are  relevant  to 
enterprise  modeling  needs  in  studying  large-scale  acquisition  programs.  Enterprise 
modeling  has  increasingly  been  incorporated  in  simulation  (Barjis,  2011;  Glazner,  2011). 

Enterprise  Simulation  and  Acquisition  Risk  Analysis 

Recent  work  has  extended  the  concept  of  organizational  simulation  to  an  enterprise 
simulation  framework  with  an  agent-based  model  for  actors  and  a  decision/event  network  to 
represent  a  risk-focused  way  of  representing  an  organizational  story  (Bodner  &  Rouse, 
2010).  Agents  represent  organizations  (e.g.,  firms  or  agencies)  that  participate  in  the 
enterprise.  They  can  perform  a  variety  of  actions  in  the  enterprise,  among  them  are  the 
following: 

•  communicate  with  other  agents, 

•  react  to  incentives  and  information, 

•  accrue  costs, 

•  change  variables  such  as  schedule  targets  or  budgets, 

•  progress  through  processes  and  tasks, 
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•  restructure  a  program,  or 

•  terminate  a  program. 

Figure  1  shows  an  example  set  of  interactions  in  an  acquisition  context.  The 
government  program  office  interacts  with  the  lead  contractor,  as  well  as  with  other  agencies. 
The  lead  contractor  manages  the  contractor  network.  Note  that  some  contractors  may  be 
sub-contracted  to  multiple  contractors.  In  a  typical  major  program,  of  course,  there  are 
dozens  of  agencies,  some  outside  the  Department  of  Defense  (DoD),  and  there  are 
thousands  of  contractors. 


Figure  1 .  Example  Interactions  of  Agents  as  Firms  or  Agencies 

In  terms  of  a  risk-focused  organizational  story,  this  framework  utilizes  a  concept  from 
interactive  drama  and  artificial  intelligence,  that  of  a  drama  manager  (Roberts  &  Isbell, 

2008).  Interactive  drama  is  a  narrative  implemented  in  computational  form,  similar  to  a 
game,  in  which  a  user  interacts  with  a  story.  The  drama  manager  interacts  with  the  story 
flow  to  give  the  story  particular  characteristics  (e.g.,  simplicity  versus  complexity).  It  does 
this  through  a  construct  known  as  a  plot  point  model,  which  abstracts  important  events  in 
the  plot  of  the  story.  A  plot  point  model  is  organized  as  a  directed  acyclic  graph  of  plot  point 
nodes,  some  of  which  may  have  precedence  relationships  between  them  (i.e. ,  arcs),  and 
some  of  which  may  be  mutually  exclusive.  A  precedence  relationship  from  node  1  and  node 
2  (i.e.,  node  1  precedes  node  2,  or  102)  implies  that  for  node  2  to  occur,  node  1  must 
occur.  Given  precedence  relationships  from  multiple  plot  points  to  a  particular  plot  point, 
only  one  of  the  predecessor  plot  points  needs  to  occur  to  allow  the  successor  plot  point  to 
occur  (i.e.,  an  “or”  relationship  among  the  precedence  relationships).  A  precedence 
relationship  can  be  mandatory  in  that  the  predecessor  must  occur  for  the  successor  node  to 
occur  (i.e.,  enabling  “and”  relationships). 

The  graph  is  acyclic  to  disallow  two  or  more  plot  points  from  simultaneously  having 
precedence  relationships  among  them.  A  terminal  plot  point  represents  an  end  to  the  story. 
The  drama  manager  may  influence  the  plot  so  that  certain  events  occur  (or  are  more  likely 
to  occur),  or  it  may  likewise  prevent  certain  events  from  occurring.  The  particular  path  of  plot 
points  realized  in  an  interactive  drama  reflects  the  instantiated  plot. 

Adapted  to  a  simulation  context,  the  plot  point  model  (or  decision/event  network)  is 
used  to  represent  important  events  in  the  simulation  that  relate,  for  example,  to  risk  in  the 
unfolding  simulation.  Thus,  the  role  of  a  drama  manager  (or  simulation  manager)  in  the 
simulation  context  is  to  guide  the  simulation  through  a  particular  set  of  decision/event  nodes 
to  reflect  a  certain  risk  profile.  The  outcome  of  the  simulation  can  then  be  compared  to  the 
risk  profile  chosen.  Figure  2  illustrates  the  relationships  among  the  simulation  manager,  the 
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plot  point  model,  and  the  simulation.  Certain  events  in  the  simulation  model  correspond  to 
decisions  or  events  in  the  decision/event  network.  However,  many  events  and  behaviors  do 
not  correspond  to  network  nodes.  Nodes  2  and  3  are  mutually  exclusive  (only  one  at  most 
may  happen).  Arcs  4->7  and  507  are  mandatory  precedence  relationships,  meaning  that 
both  nodes  4  and  5  must  occur  for  7  to  occur. 


Simulation 

Manager 


Actions 


Precedence 

Mandatory  precedence 
Mutual  exclusion 
Terminal  node 


Figure  2.  Simulation  and  Decision/Network  Interaction 

The  decision/event  network  prescribes  the  relationships  among  decisions  and 
events,  not  the  outcome  of  a  particular  organizational  story.  An  outcome  is  realized  by  a 
particular  sequence  of  decisions  and  events.  A  valid  sequence,  or  path,  must  conform  to  the 
precedence  and  exclusion  constraints  of  the  decision/event  network,  and  it  can  have  only 
one  terminal  node  (at  the  path  end).  A  particular  decision/event  network  can  generate  many 
different  paths.  An  example  valid  path  from  Figure  2  is  1— >2^6,  as  is  1— >3-*4^5— >7. 

Given  a  particular  decision/event  network,  the  enterprise  simulation  framework 
conducts  a  pre-processing  operation  to  determine  the  set  of  possible  paths,  and  the  result  is 
returned  in  the  form  of  a  partial  game  tree.  A  game  tree  is  a  construct  from  game  theory  that 
defines  moves  that  two  players  may  make  as  alternating  nodes  in  the  tree.  The  simulation 
manager  and  simulation  correspond  to  the  two  players  in  this  context.  Nodes  in  the 
decision/event  network  represent  actions  taken  by  the  simulation  (i.e. ,  decisions  and 
events).  The  simulation  manager’s  actions  are  not  represented  in  a  path,  as  these  actions 
are  intended  to  influence  the  path  followed  for  a  particular  simulation  outcome.  Hence,  the 
set  of  paths  is  a  partial  game  tree.  The  partial  game  tree  for  the  decision/event  network  is 
shown  in  Figure  3. 


Figure  3.  Partial  Game  Tree  From  Decision/Event  Network  in  Figure  2 
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The  simulation  manager’s  actions  are  based  on  its  evaluation  function.  This  function 
is  used  to  evaluate  paths  for  selection  of  a  next  node  in  the  decision/event  network  and  a 
corresponding  simulation  manager  action.  For  risk,  the  evaluation  uses  the  expected  value 
of  a  function  of  the  program’s  schedule  and  cost  outcomes  for  selection  of  the  next  node. 
This  must  be  determined  over  the  set  of  path  remainders  associated  with  each  candidate 
next  node,  where  a  path  remainder  is  the  set  of  nodes  comprising  the  rest  of  a  particular 
path  given  a  current  node  location. 

For  example,  assume  that  the  simulation  has  executed  the  partial  path  1  — »-3 — in 
Figure  3.  Nodes  5  and  6  are  the  two  candidates  to  execute  next.  The  possible  path 
remainders  are  6,  5— >6  and  6— >7.  Node  6  is  evaluated  using  simply  the  estimated  schedule 
and  cost  values  from  the  path  remainder  of  itself.  Node  5  is  evaluated  using  a  function  of  the 
schedule  and  cost  values  of  the  two  other  path  remainders. 

Of  course,  the  simulation  is  stochastic,  and  the  evaluation  can  only  use  estimates  of 
schedules  and  costs.  Another  issue  is  the  computational  complexity  associated  with  the 
evaluation  over  a  large  decision/event  network.  Thus,  the  look-ahead  capability  to  analyze  a 
path  remainder  can  be  limited  to  a  certain  number  of  nodes  in  the  path. 

The  enterprise  simulation  modeling  framework  is  implemented  using  AnyLogic™  with 
a  set  of  Java  class  extensions  for  the  agent  models,  the  simulation  manager,  plot  point 
models,  and  partial  game  trees.  AnyLogic™  is  a  commercially  available  simulation  product 
that  supports  multi-paradigm  modeling  using  discrete-event,  agent-based,  and  system 
dynamics  simulation. 

F-35  Program 

One  major  acquisition  program  that  is  of  interest  with  respect  to  risk  analysis,  and 
application  of  the  enterprise  simulation  framework,  is  the  F-35  Joint  Strike  Fighter  (JSF) 
program.  The  F-35  program  grew  out  of  the  need  for  a  next  generation  tactical  fighter  fleet 
to  replace  the  aging  fleet  of  F-1 6s  and  F-1 8s.  Due  to  the  large  size  of  these  legacy  fleets, 
the  Joint  Strike  Fighter  effort  started  as  an  ambitious,  large-scale  concept.  As  the  program 
became  more  defined,  Boeing  and  Lockheed  Martin  Aeronautics  led  consortia  that 
developed  concept  aircraft  in  a  competition  for  the  system  design  and  development  (SDD) 
contract.  When  the  Lockheed  consortium  won,  the  Joint  Strike  Fighter  program  entered  into 
SDD  in  2001.  The  initial  planned  procurement  quantity  was  2,852  planes. 

The  JSF  actually  consists  of  three  variant  aircraft  (Gertler,  2012).  The  first  (F-35A)  is 
a  conventional  take-off  and  landing  (CTOL)  aircraft  designed  for  the  Air  Force.  The  second 
(F-35B)  is  a  short  take-off  and  vertical  landing  (STOVL)  aircraft  designed  for  the  Marines. 
The  third  (F-35C)  is  a  carrier-suitable  aircraft  (CV)  designed  for  the  Navy. 

The  JSF  program  has  used  a  transformative  approach  to  acquisition  of  a  major 
weapons  system.  In  this  context,  transformation  means  a  major  change  in  the  enterprise, 
moving  from  an  as-is  enterprise  to  a  to-be  enterprise,  with  specific  intents  (Kessler  &  Heath, 
2006).  The  transformative  elements  of  this  approach  include  the  following: 

•  Rather  than  the  traditional  command-and-control  relationship  between  the  lead 
contractor  and  its  suppliers,  the  contract  required  a  partnership  model  among 
three  firms — Lockheed  Martin  Aeronautics,  BAE  Systems,  and  Northrop 
Grumman  (Bennett,  Kessler,  &  McGinnis,  in  press). 

•  The  program  was  planned  as  a  global  effort,  whereby  design,  development,  and 
production  activities  would  occur  globally  across  an  international  consortium  of 
firms,  rather  than  primarily  at  one  single  site.  Countries  with  firms  represented  in 
the  consortium  would  be  partners,  agreeing  to  purchase  F-35s  (Kapstein,  2004). 
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•  Modeling  and  simulation  technologies  have  been  used  extensively  in  SDD, 
essentially  to  provide  a  “testing  in  software”  capability.  The  confidence  in  this 
testing  approach  enabled  substantial  concurrency  to  be  designed  into  the 
program  between  development  and  production  so  that  production  started  prior  to 
the  completion  of  flight  testing  (DoD,  201 1 ).  The  level  of  concurrency  was  higher 
than  in  similar  programs. 

•  The  program  is  developing  three  variants  using  a  common  platform  targeting  a 
high  level  of  commonality  of  components  across  the  three  variants.  The  intent  is 
to  reduce  cost  (promote  affordability)  while  providing  aircraft  to  support  three 
mission  types.  The  three  mission  types  require  different  high-level  technical 
capabilities  (stealth,  STOVL,  supersonic,  and  carrier  capabilities)  that  historically 
have  not  been  combined  to  this  degree  in  a  single  craft  or  platform  (Blickstein  et 
al. ,  2011). 

•  The  production  supply  network  was  planned  for  use  by  sustainment  (Bennett  et 
al.,  in  press). 

One  of  the  first  efforts  in  the  SDD  phase  was  to  set  up  the  global  technology 
infrastructure  needed  to  support  the  design  of  the  F-35  among  the  different  firms  involved. 
This  was  a  substantive  and  transformative  initiative  that  required  significant  alignment 
among  the  enterprise  leadership  and  major  changes  to  processes  among  all  stakeholder 
firms  (Bodner  et  al.,  2011).  The  technology  requirements  were  challenging — providing  real¬ 
time  global  access  to  the  design  database  subject  to  myriad  management  and  security 
constraints.  The  technical  challenges  turned  out  to  be  less  difficult  than  the  social  and 
process  changes  needed.  Overall,  the  setup  of  the  technology  infrastructure  was  deemed 
successful. 

Nevertheless,  such  major  transformative  elements  entail  risk  (Rouse,  2006),  and 
there  have  been  a  number  of  less-than-desirable  outcomes  (and  potential  future  outcomes) 
for  the  program. 

•  A  reduction  in  planned  purchases  of  400  units  occurred  early  in  SDD,  driving  up 
unit  costs  (Blickstein  et  al.,  201 1 ). 

•  In  the  earlier  part  of  SDD,  there  were  weight  and  design  issues  that  required 
significant  additional  effort  to  address  (Blickstein  et  al.,  2011). 

•  The  weapons  bay  of  the  CV  variant  was  redesigned  for  larger  payload,  resulting 
in  the  same  redesign  for  the  STOVL  and  CTOL  variants.  This  redesign  caused 
weight  and  stability  problems  for  the  STOVL  (Blickstein  et  al.,  2011). 

•  Due  to  schedule  slippages  and  cost  growth,  the  program  was  re-baselined  in 
2004  and  2007  (Blickstein  et  al.,  2011). 

•  After  the  DoD’s  Joint  Estimating  Team  issued  a  report  in  late  2009  stating  that 
the  program  would  need  an  additional  30  months  to  complete  SDD,  the  program 
was  restructured  (GAO,  2011),  adding  13  months  to  the  SDD  schedule  (as  well 
as  the  needed  funding),  and  withholding  $614  million  in  award  fees  from  the  lead 
contractor.  Three  aircraft  were  also  added  to  early  production. 

•  In  2010,  the  program  was  found  to  have  increased  in  cost  over  both  the  original 
baseline  and  the  then-current  baseline  by  enough  that  it  was  certified  with  a 
Nunn-McCurdy  breach  (Blickstein  et  al.,  2011). 

•  Continued  technical  issues  with  the  STOVL  variant  resulted  in  that  part  of  the 
program  being  placed  on  a  two-year  “probation,”  with  its  production  schedule 
moved  back  (GAO,  201 1 ). 
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•  The  U.S.  government’s  fiscal  situation  has  called  into  question  whether  the 
planned  quantities  of  aircraft  will  be  purchased.  Already  the  U.K.  government  has 
reduced  its  planned  purchases  (Gertler,  2012). 

Blickstein  et  al.  (2011)  conducted  a  root  cause  analysis  of  the  Nunn-McCurdy  breach 
and  identified  the  following  underlying  risk  factors  as  root  causes: 

•  SDD  was  designed  around  a  concept  demonstrator  rather  than  around  a 
prototype  that  would  have  accounted  better  for  producibility. 

•  The  engineering  strategy  focused  on  developing  the  CTOL  variant  first,  rather 
than  tackling  the  more  difficult  engineering  design  and  development  of  the 
STOVL. 

•  The  original  baseline  was  overly  optimistic  on  cost  and  schedule  due  to 
assumptions  on  component  commonality  and  technology  integration. 

These  in  turn  drove  other  issues  that  resulted  in  the  eventual  breach.  Getting  the 
high-level  technical  capabilities  across  the  variants  was  more  technically  challenging  than 
anticipated,  involving  extensive  trade-offs.  The  initial  weight  problems  caused  redesign,  and 
these  design  changes  had  to  be  distributed  throughout  the  enterprise  of  collaborators.  Due 
to  the  large  number  of  suppliers,  this  process  delayed  test  and  production  and  hence  drove 
up  costs.  Delays  also  resulted  in  work  being  done  with  higher  labor  rates. 

The  issue  of  concurrency  has  been  raised  as  a  cause  of  the  program’s  cost  and 
schedule  issues  (DoD,  2011).  Historical  analysis  has  indicated  that  concurrency  has  not 
been  a  major  cost  and  schedule  risk  driver  (Congressional  Budget  Office  [CBO],  1988). 
However,  this  type  of  analysis  likely  does  not  account  for  the  complexity  of  the  JSF  program. 

At  present,  the  program  is  entering  a  fifth  low-rate  initial  production  (LRIP)  increment 
that  is  a  fixed-price  contract  with  risk  sharing  between  the  government  and  contractor 
(Gertler,  2012). 

Simulation  Model 

An  example  enterprise  simulation  model  and  associated  decision/event  network  for 
the  SDD  portion  of  the  F-35  program  (and  overlap  with  low-rate  initial  production)  is  now 
presented.  This  model  mainly  addresses  the  U.S. -funded  portion  of  the  JSF  program. 
Foreign  partner  purchases  are  treated  as  exogenous  effects. 

The  outcomes  of  interest  in  the  simulation  model  relate  to  schedule  and  cost.  Let  /  be 
an  index  for  fiscal  years  since  the  start  of  SDD  in  October  of  2001 .  Thus,  /  =  1  corresponds 
to  FY2002,  /  =  2  to  FY2003,  and  so  on.  Let  j  likewise  be  an  index  for  fiscal  years.  Let  k  index 
the  variants,  with  k  =  1  representing  the  CTOL,  k  =  2  representing  the  STOVL,  and  k  =  3 
representing  the  CV.  Finally,  let  /  be  an  index  for  program  milestones,  with  /  =  1  representing 
preliminary  design  review  (PDR),  /  =  2  representing  critical  design  review  (CDR)  for  the 
CTOL  variant,  /  =  3  representing  CDR  for  the  STOVL  variant,  /  =  4  representing  CDR  for  the 
CV  variant,  /  =  5  representing  flight  test  start  for  the  CTOL  variant,  /  =  6  representing  flight 
test  start  for  the  STOVL  variant,  /  =  7  representing  flight  test  start  for  the  CV  variant,  /  =  8 
representing  flight  test  finish  for  the  CTOL  variant,  /  =  9  representing  flight  test  finish  for  the 
STOVL  variant,  /  =  10  representing  flight  test  finish  for  the  CV  variant,  /  =  1 1  representing 
start  of  the  first  LRIP  increment,  /  =  12  representing  start  of  the  second  LRIP  increment,  /  = 
13  representing  start  of  the  third  LRIP  increment,  /=  14  representing  start  of  the  fourth  LRIP 
increment,  /  =  15  representing  start  of  the  fifth  LRIP  increment,  and  /  =  16  representing  start 
of  production  ramp-up. 
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The  following  state  variables  for  the  program’s  schedule  and  cost  targets  are 
defined.  Note  that  the  target  values  can  change  from  year  to  year,  which  occurs,  for 
example,  when  the  schedule  targets  or  planned  purchases  are  moved  back. 

Cij  =  cost  target  in  year  /  for  development  costs  in  year  j  ($) 

Qjjk  =  planned  quantities  in  year  /  for  procurement  in  year  j  of  variant  k  (number  of  craft) 
Uijk  =  per-unit  cost  target  in  year  i  for  procurement  in  year  j  of  variant  k  ($) 

Su  =  schedule  target  in  year  /  for  milestone  /  (number  of  months  since  SDD  kick-off) 

Li  =  contract  cost  for  LRI P  increment  /  (/  =  11,  12  ...  15) 

State  variables  for  program  status  are  needed.  Let  q  refer  to  the  original  target  for 
the  status  variable,  r  to  the  current  target,  s  to  the  current  progress,  and  t  to  the  current  time 
(since  beginning  of  SDD).  Finally,  let  u  denote  an  index  for  capabilities,  with  u  =  1  being 
stealth,  u  =  2  being  supersonic  speed,  u  =  3  being  STOVL  capable,  and  u  =  4  being  carrier 
capable.  The  following  variables  are  defined. 

Gq  =  original  target  for  completion  effort  of  global  design  capability  =  100% 

Gn  =  current  target  for  completion  effort  of  global  design  capability  at  time  t  (%  of  Gq) 

Gst  =  current  progress  toward  completion  of  global  design  capability  at  time  t  (%  of  Gq) 

Dq  =  original  target  for  completion  effort  of  engineering  design  files  =  100% 

Drf  =  current  target  for  completion  effort  of  engineering  design  files  at  time  t  (%  of  Dq) 

Dst  =  current  progress  toward  completion  of  engineering  design  files  at  time  t  (%  of  Dq) 

Kq  =  original  target  for  completion  effort  of  software  =  100% 

Krt  =  current  target  for  completion  effort  of  software  at  time  t  (%  of  Kq) 

Kst  =  current  progress  toward  completion  of  software  at  time  t  (%  of  Kq) 

F kq  =  original  target  for  completion  effort  of  flight  testing  for  variant  k  =  100% 

F krt  =  current  target  for  completion  effort  of  flight  testing  for  variant  k  at  time  t  (%  of  Fkq) 
Fkst  =  current  progress  toward  completion  of  flight  testing  for  variant  k  a  time  t  (%  of  Fkq) 
Mkqu  =  original  target  for  completion  effort  of  capability  u  for  variant  k  =  100% 

Mkrut  =  current  target  for  completion  effort  of  capability  u  for  variant  k  at  time  t  (%  of  Mkqu) 
Mksut  =  current  progress  toward  completion  of  capability  u  for  variant  k  a  time  t  (%  of 
Mkqu) 

Piq  =  original  target  for  completion  effort  of  LRIP  increment  /  (/  =  11,  12  ...  15)  =  100% 

Pirt  =  current  target  for  completion  effort  of  LRIP  increment  /  (/  =  11,  12  ...  15)  at  time  t  (% 
of  P*,) 

Pist  =  current  progress  toward  completion  of  LRIP  increment  /  (/  =  11,  12  ...  15)  at  time  t 

(%  of  P,q) 

Wkt  =  current  weight  of  variant  k  as  a  percentage  of  the  maximum  weight  target  at  time  t 

(%) 

E  =  level  of  concurrency  designed  into  program 

Ht  =  current  level  of  component  commonality  among  variants  at  time  t  (%) 

At  =  SDD  cost  accrual  rate  at  time  t  ($  per  time  unit) 

Vt  =  cumulative  SDD  cost  accrued  through  time  t  ($) 

B/t  =  cost  accrual  rate  for  LRIP  increment  /  (/  =  1 1,  12  ...  15)  at  time  t  ($  per  time  unit) 

Xn  =  cumulative  cost  for  LRIP  increment  /  (/  =  11,  12  ...  15)  through  time  t  ($) 

Note  that  Gq  is  a  constant  and  reflects  an  original  completion  effort  level  of  100%. 

The  actual  work  to  be  done  for  completion,  Gn,  may  increase  (or  decrease)  over  time.  The 
percentage  toward  actual  completion  is  thus  Gsf/Grt.  The  same  model  is  used  for  engineering 
design  files,  software,  capability  achievement,  flight  testing,  and  LRIP  completion.  It  should 
be  noted  for  the  capability  completion  variables  that  variables  exist  only  for  valid 
combinations  of  variant  k  and  capability  u  (i.e. ,  there  is  no  variable  for  the  CTOL  having  a 
carrier  capability). 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  81  - 


The  actor  model  is  relatively  simple  and  consists  of  a  government  agent  and  a 
contractor  agent.  The  government  agent  oversees  the  program,  and  the  contractor  agent 
executes  the  work.  The  government  can  make  decisions  such  as  program  restructure  or 
program  cancellation.  The  contractor  has  parameters  that  govern  its  progress  rate  toward 
completion  of  the  various  efforts  needed  for  the  program  (e.g.,  software).  The  contractor 
also  has  an  alignment  parameter  that  reflects  the  extent  to  which  its  executive  leadership 
and  technical  management  across  the  enterprise  are  aligned  with  a  strategy  to  stand  up  the 
global  design  technology  infrastructure.  This  parameter  affects  the  rate  at  which  that 
capability  is  achieved.  In  future  versions  of  the  model,  the  contractor  agent  will  be  replaced 
with  multiple  agents  representing  different  contractor  firms,  and  the  alignment  parameter  will 
be  refined  so  that  individual  firm  alignments  with  enterprise  strategy  are  modeled. 

The  decision/event  model  is  shown  in  Figure  4.  On  the  right  side  of  the  figure  are 
various  program  milestones  (e.g.,  design  reviews,  flight  testing,  and  production).  The  left 
side  focuses  largely  on  decisions  made  by  the  government  and  contractor  agents  that 
influence  program  execution  (e.g.,  program  design),  as  well  as  on  exogenous  events  (e.g., 
reduction  in  planned  foreign  partner  purchases).  The  decisions  on  the  left  side  reflect  a 
number  of  risk  drivers  in  JSF  acquisition. 
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Figure  4.  JSF  SDD  Decision/Event  Network 

Nodes  19-23  represent  program  design  decisions,  and  they  are  grouped  to  reflect 
that  they  must  be  preceded  by  contract  award,  and  that  they  precede  preliminary  design 
review  (PDR).  The  mutual  exclusion  constraints  mean  that  only  one  of  nodes  20  and  21  and 
one  of  nodes  22  and  23  would  occur  to  enable  node  25.  Nodes  26-32  represent  post-PDR 
program  management  decisions.  Except  for  node  29  (which  cannot  occur  until  after  the  first 
LRIP),  they  are  grouped  to  reflect  this.  Table  1  describes  the  various  plot  points  in  more 
detail. 
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Table  1.  Decision/Event  Descriptions 


Decision/Event  Node 

Description 

1 .  Contract  Awarded 

SDD  contract  awarded  to  Lockheed  Martin  consortium.  This 
initiates  the  organizational  story. 

2.  Global  Design  Capability 

The  enterprise  achieves  capability  to  support  JSF  design  via 
global  technology  infrastructure. 

3.  PDR 

The  program  passes  preliminary  design  review. 

4.  CTOL  CDR 

The  CTOL  variant  passes  critical  design  review.  Note  that 

CDR  is  a  separate  event  for  each  variant. 

5.  STOVL  CDR 

The  STOVL  variant  passes  critical  design  review. 

6.  CV  CDR 

The  CV  variant  passes  critical  design  review. 

7.  CTOL  Flight  Test  Start 

The  CTOL  variant  starts  flight  testing.  Once  again,  flight  tests 
are  modeled  separately  for  each  variant. 

8.  STOVL  Flight  Test  Start 

The  STOVL  variant  starts  flight  testing. 

9.  CV  Flight  Test  Start 

The  CV  variant  starts  flight  testing. 

10.  CTOL  Flight  Test  Finish 

The  CTOL  variant  finishes  flight  testing. 

11.  STOVL  Flight  Test 

Finish 

The  STOVL  variant  finishes  flight  testing. 

12.  CV  Flight  Test  Finish 

The  CV  variant  finishes  flight  testing. 

13.  LRIP1 

The  first  low-rate  initial  production  contract  is  initiated.  The 
details  (cost  and  number  of  each  variant)  are  captured  in  the 
simulation. 

14.  LRIP2 

The  second  low-rate  initial  production  contract  is  initiated. 

15.  LRIP3 

The  third  low-rate  initial  production  contract  is  initiated. 

16.  LRIP4 

The  fourth  low-rate  initial  production  contract  is  initiated. 

17.  LRIP5 

The  fifth  low-rate  initial  production  contract  is  initiated.  The 
current  model  assumes  only  five  LRIPs.  In  future  versions, 
the  number  will  be  variable,  with  the  simulation  having  the 
capability  to  generate  LRIPs. 

18.  Production  Ramp-up 

Production  ramp-up  begins,  ending  SDD.  This  is  a  terminal 
decision/event. 

19.  Prototype  Development 

This  represents  the  program  enterprise  decision  to  develop  or 
not  develop  a  prototype  aircraft  in  addition  to  the  concept 
aircraft  that  won  the  SDD  contract.  Note  that  an  affirmative 
decision  delays  PDR. 

20.  CTOL-First  Strategy 

The  program  decides  to  design  the  CTOL  variant  first. 

21.  STOVL-First  Strategy 

The  program  decides  to  design  the  STOVL  variant  first.  Note 
that  this  is  mutually  exclusive  with  the  CTOL-first  strategy. 

22.  High  Concurrency 

The  program  designs  a  high  level  of  concurrency  into  the 
program.  The  baseline  case  assumes  moderate  concurrency. 

23.  No  Concurrency 

The  program  designs  no  concurrency  into  the  program.  Note 
that  this  is  mutually  exclusive  with  the  high  concurrency 
decision. 

24.  Collaborative  Alignment 

The  enterprise  achieves  or  does  not  achieve  collaborative 
alignment  on  the  changes  needed  to  enable  the  global  SDD 
technology  infrastructure  to  be  stood  up.  Achievement 
includes  executive  alignment  among  partner  organizations, 
as  well  as  technical  management  alignment.  Achievement 
speeds  up  technology  development  over  the  case  where 
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there  is  little  alignment. 

25.  Program  Design 

Program  design  is  complete. 

26.  Weight  Reduction 

The  program  engages  in  a  major  weight  reduction  effort. 

27.  Nunn-McCurdy  Breach 

A  Nunn-McCurdy  breach  is  declared. 

28.  Program  Restructure 

The  program  is  restructured,  changing  cost  and  schedule 
targets. 

29.  Fixed-Cost  Contract 

The  contract  moves  from  cost  plus  to  fixed  price.  It  is 
assumed  that  this  cannot  occur  until  after  the  first  LRIP. 

30.  Major  Planned  U.S. 
Purchase  Reduction 

The  U.S.  decides  to  reduce  its  purchase  quantity. 

31.  Major  Planned  Foreign 
Purchase  Reduction 

Foreign  partners  decide  to  reduce  their  purchase  quantity. 

32.  Program  Canceled 

The  government  cancels  the  program. 

Risk  Analysis  and  Mitigation 

Current  work  is  addressing  detailed  analysis  of  risk  in  the  JSF  SDD  phase  using  the 
enterprise  simulation  model.  This  section  describes  the  analysis  being  done. 

1 .  One  set  of  analysis  involves  the  effect  of  having  a  high  degree  of  alignment 
among  enterprise  stakeholders  for  the  strategy  needed  to  stand  up  the  global 
design  technology  infrastructure  needed  to  support  design  efforts  across  the 
enterprise.  Without  alignment,  this  capability  is  delayed  (thus  delaying  initial 
design  activities)  and  may  not  be  as  effective  as  with  better  alignment, 
resulting  in  downstream  design  issues  that  must  be  addressed  (causing 
increased  cost  and  delayed  schedules). 

2.  Another  set  of  analysis  involves  program  design  decisions.  High  levels  of 
concurrency  can  shorten  the  time  until  systems  are  produced  and  deployed, 
thus  helping  meet  schedule  targets.  However,  undiscovered  issues  may 
affect  production  and  cause  rework,  thus  negatively  affecting  cost  and 
schedule.  Low  levels  of  concurrency  on  the  surface  extend  the  schedule 
beyond  what  is  possible  with  higher  levels,  but  have  less  risk  from  design 
issues.  Similarly,  producing  a  prototype  system  delays  design  of  the  real 
system.  Is  this  compensated  for  by  improvements  in  schedule  and  cost  of 
downstream  activities? 

3.  Finally,  what  is  the  effect  of  exogenous  events  such  as  major  reductions  in 
planned  purchases?  This  tends  to  drive  up  per-unit  costs,  making  future 
purchases  more  costly  and  less  likely  to  occur. 

Conclusion  and  Future  Research 

This  paper  has  presented  a  novel  enterprise  simulation  approach  for  analyzing  risk  in 
acquisition  programs.  Risk  continues  to  be  a  major  concern  in  acquisition,  especially  as 
program  budgets  are  under  increasing  pressure  from  fiscal  constraints.  Simulation  is 
appealing  to  use  in  risk  analysis  since  it  enables  probabilistic  analysis  of  system  and 
organizational  behavior  and  outcomes. 

The  enterprise  simulation  approach  differs  from  traditional  simulation  in  two 
important  ways.  First,  traditional  simulation  approaches  focus  on  analyzing  the  technical 
aspects  of  system  and  organizational  behaviors.  Increasingly,  sociotechnical  phenomena 
are  important  to  the  success  and  failure  of  large  programs  due  to  their  nature  as  enterprises. 
For  instance,  the  level  of  alignment  among  the  JSF  enterprise  actors  as  to  a  strategy  for 
standing  up  the  global  design  technology  infrastructure  is  an  important  sociotechnical 
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behavior  that  influences  the  time  needed  for  this  activity,  as  well  as  the  success  of  the 
outcome. 

Second,  the  enterprise  simulation  approach  explicitly  models  a  decision/event 
network  that  can  be  used  to  link  risk  drivers  with  downstream  program  activities  and 
outcomes.  Traditional  simulation  approaches  tend  to  model  this  implicitly,  so  that  it  may  not 
be  effectively  modeled,  and  also  so  that  the  organizational  story  (path  realization)  is  not 
explicitly  presented. 

Future  work  involves  continued  elaboration  and  validation  of  the  model,  as  well  as 
analysis  of  particular  risks. 
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Motivation 


•  Many  enterprises  are  in  need  of  transformation 
due  to  challenges  or  opportunities 

•  Defense  acquisition  is  one  such  enterprise 

-  Fiscal  challenges 

-  Aging  system  portfolio 
—  New  system  complexity 

—  New  and  emerging  threats 

•  Enterprise  transformation  entails  risk 

•  Computational  models  such  as  simulation  are  one 
way  to  discover,  understand  and  mitigate  risk 
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Enterprise  Transformation  Framework 


Rouse,  Enterprise  Transformation,  2006 


Transformation  efforts  in 
the  green  and  yellow  circles 
are  less  risky,  but  only  have 
limited  pay-offs 

Efforts  in  the  orange  and 
red  circles  are  enormously 
risky,  but  can  change  the 
whole  market 

Successful  transformations 
in  the  orange  and  red  circles 
typically  require 
competitors  to  respond  at 
least  with  efforts  in  the 
green  and  yellow  circles 
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Enterprise  Simulation 

•  Traditional  simulation 

—  Analysis  of  behavior  and  performance  over  time 
under  different  scenarios 

-  Uncertainty  incorporated 

—  Primary  emphasis  on  processes 

•  Enterprise  simulation 

—  Socio-technical  phenomena  modeled 

—  Co-emphasis  on  firm  or  individual  behavior  via  agent- 
based  simulation 

—  Representations  of  organizational  behavior 
—  Risk  drivers  modeled 
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Agent-Based  Organizations 


Other  DoD  Agencies 


Communicate  with 
other  agent 
organizations 

Accrue  costs 

Dispense  funds 

Progress  through 
processes  or  tasks 

Restructure  a  program 

Terminate  a  program 
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Acquisition  Processes 


•  Decision  points 

•  Cost  accruals  by  agent 
organizations 

•  Progress  through 
acquisition  milestones 

—  Program  design 
—  Program  reviews 

-  Development/test 

-  LRIPs 

•  Concurrency  and 
uncertainty 

•  Contract  renegotiations 


SDD 

Program 

Flight  Test 

LRIPs 

Production 

Initiation 

Reviews 

Ramp-Up 

(PDR,  CDR) 

« - M - • - • - • - ► 


Possible  contract  renegotiations 
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Decision/Event  Network 


*  Precedence 


Mandatory  precedence 
Mutual  exclusion 
Terminal  node 


Directed  acyclic  graph  of  precedence 
relationships  between  important 
nodes 

-  Decisions  by  agents 

-  Events  in  simulation 

Important  events  abstracted  from 
simulation  model 

Concept  of  simulation  manager 
borrowed  from  interactive  drama 

Simulation  manager  seeks  to  guide 
the  simulation  through  a  "path"  of 
decision/event  nodes 

-  Receive  state  information  from 
simulation 

-  Takes  actions  to  influence 
determination  of  next  node  in 
decision/event  network 

-  Actions  based  on  evaluation  function  of 
future  paths  from  candidate  next  nodes 

-  Evaluation  function  based  on  a  certain 
goal  for  the  outcome 


5/16/2012 


2012  Acquisition  Research  Symposium 


8 


Partial  Game  Tree 


Potential  Outcomes 
•  1  ->  2  ->  6 
•l->3->4->5->6 
•1^3^4^5^7 
•1^3^4^6 
•1^3^5^4^6 
•1^3^5^4^7 


•  Enumeration  of  possible 
event  paths  and 
outcomes 

•  Generated  from 
decision/event  network 

•  "Partial"  game  tree  due  to 
simulation  manager 
actions  not  being 
represented 

•  Computational 
complexity  is  an  issue  for 
large  simulations 
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Evaluation  Function  for  Risk 


•  Risk  is  one  outcome  type  of 
interest 

•  Risk  is  the  expected  value  of  an 
undesired  outcome 

•  Schedule  and  cost  are  two  types 
of  risk 

•  Schedule  risk 

-  Expected  additional  schedule  growth  S 

•  Cost  risk 

-  Expected  additional  cost  growth  C 

•  Evaluation  uses 

-  Estimates  of  S  and  C  along  each 
remaining  path 

-  Probabilistic  weightings  for  each  path 

-  Initial  estimates  and  weightings  are 
used  then  are  updated  as  simulation 
progresses 


path  1 
path  2 


path  3 


•  Assuming  current  node  is  i: 

2 

Z/V  /V 

plSlp2S2 

1  r= 1 

/\ 

-  Compute  Ek(S)  =  S 3 

-  Execute  simulation  manager  to  prefer 
one  (based  on  risk  analysis  preference) 
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Simulation  Implementation 


Simulation 

Manager 


k  Actions 

\ 

\ 


Implemented  using  AnyLogic 

•  Commercial  simulation  tool 

•  Supports  multi-paradigm  simulation 

•  Java-extensible 
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F-35  Program  Background 

•  Next  generation  replacement  for  tactical  fighter  fleets 
(F-16,  F-18) 

•  Concept  aircraft  competition  between  consortia  led  by 
Lockheed  Martin  Aeronautics  and  Boeing 

•  SDD  contract  awarded  to  LM  Aeronautics  consortium 
in  2001 

•  Three  variants  envisioned: 

—  Conventional  take-off  and  landing  (CTOL)  -  Air  Force 
—  Carrier  variant  (CV)  -  Navy 

—  Short  take-off  and  vertical  landing  (STOVL)  -  Marines 

•  2,852  total  U.S.  aircraft  planned  for  procurement 

•  Schedule  and  cost  issues 
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F-35  Transformative  Risk  Elements 


Traditional  Approach 

F-35  Approach 

Single  lead  firm  with  command-and- 
control  business  model 

Collaborative  partnership  model  among 
three  lead  firms  (LM  Aero,  BAE,  NG) 

U.S.-based  design,  development  and 
production  with  foreign  sales  as  secondary 
consideration 

Global  design,  development  and 
production  with  foreign  sales  as  major 
strategy 

Most  testing  done  with  real  articles; 
limited-to-medium  concurrency 

Substantial  testing  done  with  modeling  & 
simulation;  high  concurrency 

Two  technically  complex  capabilities  per 
aircraft  at  most 

•  F-22  (stealth,  supersonic) 

•  F-14  (supersonic,  carrier-capable) 

Three  variants  on  a  common  platform  with 
four  technically  complex  capabilities  across 
the  family 

•  Stealth,  supersonic,  STOVL,  carrier- 
capable 

Separately  managed  production  and 
sustainment  networks 

Combined  production  and  sustainment 
networks 
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Transformation  Map  -  Partnership 


Collaborative 

partnership 

model 


MEANS 


Collaborative 

partnership 

model 
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Transformation  Map -Global  Program 


MEANS 


Global  design, 
development, 
production  & 
sales 


Global  design, 


development. 


production  & 


sales 
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Transformation  Map -Testing 


MEANS 


Model-based 
testing  and 
concurrency 


Model-based 
testing  and 
concurrency 


Model-based 
testing  and 
concurrency 


ENDS 
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Transformation  Map  -  Capabilities 
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Transformation  Map -Supply  Network 


Combined 
production  & 
sustainment 
networks 


Combined 
production  & 
sustainment 
networks 


Combined 
production  & 
sustainment 
networks 


MEANS 


SCOPE 


ENDS 
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F-35  Decision/Event  Network 


1 .  Contract 
Awarded 
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Development 
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Reduction 


28.  Program 
Restructure 
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Planned 
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Purchase 

Reduction 


27.  Nunn- 
McCurdy 
Breach 


31 .  Major 
Planned 
Foreign 
Purchase 
Reduction 


32.  Program 
Canceled 
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24.  Collaborative 
Alignment 


4.  CTOLCDR 


7.  CTOL 
Flight  Test 
Start 
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10.  CTOL 
Flight  Test 
Finish 


13.  LRIP1 
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2.  Global 
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Design 
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5.  STOVL 
CDR 


8.  STOVL 
Flight  Test 
Start 


1 1 .  STOVL 
Flight  Test 
Finish 


6.  CVCDR 


9.  CV  Flight 
Test  Start 


29.  Fixed  Cost 
Contract 


14.  LRIP2 


15.  LRIP3 


16.  LRIP4 


17.  LRIP5 
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12.  CV  Flight 
Test  Finish 


18.  Production 
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Simulation  Model 

•  Government  and  lead  contractor  modeled  as 
agents 

•  SDD  and  LRIP  phases  of  acquisition  processes 

•  Simulation  variables  track: 

-  Cost  targets 

-  Schedule  targets 

—  Progress  on  milestones 
—  Status  of  weight  TPM 
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Analysis 

•  Simulation  has  been  run  for  a  subset  of  the 
decision/event  network 

—  Collaborative  alignment 

-  Program  design  -  concurrency  and  prototype 

-  Technical  aspects  -  design  capability,  reviews,  test 

-  LRIPs 

•  Cost  risk  and  schedule  risk  have  been  assessed 

•  Since  many  probability  parameters  are  unknown, 
sensitivity  analysis  is  being  conducted  to 
determine  sensitivity  over  ranges 
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Results  to  Date 


•  Achieving  collaborative  alignment  by 
leadership  is  critical  to  risk  reduction 

•  High  concurrency  has  a  higher  schedule  and 
cost  risk  than  no  concurrency 

-  Without  concurrency,  schedule  and  cost  targets 
cannot  be  met 

—  With  concurrency,  risk  of  not  meeting  schedule 
and  incurring  additional  costs 

•  Prototype  reduces  schedule  and  cost  risk 
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Conclusions 


•  Enterprise  transformation  analysis  suggests  that 
the  F-35  program  was  enormously  risky 

•  Enterprise  simulation  is  used  as  a  way  to 
understand  risks  by  modeling  risk  drivers 

•  Results  to  date  indicate 

—  Alignment  is  critical  reduce  risk  associated  with 
collaboration 

—  Concurrency  leads  to  increased  risk  (sensitivity 
analysis  needed  to  improve  characterization) 

-  A  prototype  of  a  complex  system  can  lead  to  reduced 
risk  (sensitivity  analysis  needed  to  improve 
characterization) 
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Questions? 
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